Viewer tracking image display

ABSTRACT

Image information displayed on an electronic device can be adjusted based at least in part upon a relative position of a viewer with respect to a device. In some embodiments, image stabilization can be provided such that an image remains substantially consistent from the point of view of the viewer, not the display element of the device. The image can be stretched, rotated, compressed, or otherwise manipulated based at least in part upon the relative viewing position. Similarly, the viewer can move relative to the device to obtain different views, but views that are consistent with the viewer looking at an object, for example, through a piece of glass. The device can overlay information on the image that will adjust with the adjusted image. Three-dimensional modeling and display can be used to offset parallax and focus point effects.

BACKGROUND

As the capabilities of various computing devices increase, and as peopleare utilizing computing devices for an increasing variety of tasks,there are additional challenges being faced in meeting userexpectations. For example, an increasing number of applications areproviding three-dimensional, virtual reality, or enhanced realityexperiences. In one example, it can be desirable to overlay varioustypes of information over an image being captured and/or displayed on acomputing or electronic device. If the device is a portable device, theuser's hand might jitter which can cause the image being displayed tomove with respect to the user, which can take away from the virtualexperience. Further, if the user changes position or rotates the device,the display of the device will not adjust accordingly, such that therealistic nature of the display can be significantly degraded.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an environment in which a user is able to glance atan electronic device to view captured image information in accordancewith various embodiments;

FIGS. 2( a), 2(b), and 2(c) illustrate example displays that can begenerated in accordance with various embodiments;

FIGS. 3( a)-3(b) illustrate example displays that can be generated inaccordance with various embodiments;

FIGS. 4( a)-4(d) illustrate example orientations that can be addressedin accordance with various embodiments;

FIG. 5 illustrates front and back views of an example of a computingdevice that can be used in accordance with various embodiments;

FIG. 6 illustrates example components of a computing device such as thatillustrated in FIG. 5;

FIG. 7 illustrates an example process for determining relative positionthat can be used in accordance with various embodiments;

FIG. 8 illustrates an example process for managing the display ofcontent in response to relative position that can be used in accordancewith various embodiments;

FIGS. 9( a)-9(f) illustrate example approaches to determining headposition and/or gaze direction that can be used in accordance withvarious embodiments;

FIGS. 10( a)-10(b) illustrate example approaches to determining changesin the relative distance to a user in accordance with variousembodiments; and

FIG. 11 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the aforementioned andother deficiencies experienced in conventional approaches to displayingcontent on an electronic device. In particular, various embodimentsenable adjustments to be made to displayed image (i.e., still or video)content based at least in part upon a current relative position of aviewer with respect to the device.

In various embodiments, an image stabilization algorithm providesstability of a captured video image so the image is stable not withrespect to a display element, as in conventional approaches, but withrespect to the viewer. While many image stabilization algorithms forapplications such as digital video cameras attempt to remove jitter fromthe captured image so that an object centered in the field of viewremains centered in the image, algorithms discussed herein insteadattempt to have that item remain in substantially the same locationrelative to the position of the viewer. If the device is displayingimage information captured from the other side of the device, forexample, the device can act like a virtual piece of glass, wherein thelocation and/or orientation of an object being viewed by a user willremain substantially unchanged regardless of the relative orientation ofthe device.

Similarly, a user can change the position of that user's head to viewimage information from different angles as displayed by the device. Forexample, a device might have a camera on the back that, by default,points in a direction substantially orthogonal to the primary plane ofthe device. In many conventional devices, the displayed portion of thecaptured image depends upon the orientation of the camera, and not upona relative position of the viewer. In embodiments discussed herein,however, the portion of the captured image that is displayed can dependat least in part upon the relative position of the viewer with respectto the device. For example, if the viewer leans to the right, thedisplayed image might “pan” to the left such that the device still actslike a pane of glass. The viewer thus is able to change what isdisplayed by the device at least in part by moving relative to thedevice.

In some embodiments, the device might not act like a piece of glass, butmight instead provide different views based upon the relative positionof the user to the device. For example, a user might get a camera viewto scroll in one direction by leaning in another. Further, the usermight get a three-dimensional image to rotate by changing the user'srelative position in a specific direction.

In various embodiments, an electronic device is capable of overlayinginformation on an image to be displayed. For example, the device mightanalyze the image and display pertinent information on the image. Inother embodiments, the device might enable a user to place graphicalelements over specific objects in the image. Various other types ofcontent can be added as well as discussed elsewhere herein. Using therelative position of the user, this additional content can move “with”the associated object in the image, such that the association of thecontent with the object is not disturbed based on movement of the deviceand/or viewer.

The image information displayed can be information captured by at leastone imaging element of the same electronic device or a different device.Further, the image information can be information rendered by the sameor a different electronic device, such as may be part of a mediapresentation or video game.

In at least some embodiments, at least two cameras are used to detectthe relative position of the viewer and/or capture the image informationto be displayed. Using two cameras to capture either type of informationenables a three-dimensional determination of relative position to bemade. In at least some embodiments, a three-dimensional model can begenerated to assist in determining the proper image view, portion,and/or perspective to display. Orientation-determining elements such asaccelerometers or gyroscopes also can be used to assist in determiningchanges in the relative orientation of the user.

In various embodiments, an image manipulation algorithm also can beutilized to stretch, compress, rotate, or otherwise manipulate the imagesuch that the image appears of a consistent perspective to the user,such as may exhibit a normal aspect ratio, regardless of the relativeposition of the user. For example, a viewer looking at a person“through” the device can rotate the device without the apparentperspective or aspect ratio of that other person changing, from thepoint of view of the viewer.

The relative position of specific features of a viewer can also betracked by the device. For example, a device can monitor the currentlocation of at least one specified feature of the viewer, such as theviewer's eyes or mouth, and can adjust the determined relative locationaccordingly. Thus, if the viewer moves or the device is moved such thatthe relative position between the viewer and the device is altered, thedevice can adjust input determination accordingly, such as to accountfor the new relative position and prevent false input.

In some embodiments, the device might have elements that are capable ofmoving or rotating to provide directionality. For example, an imagecapture element might include components such as a moving camera,adjustable mirror or lens, rotating gimbal, or other such element toincrease the field of view and/or enable image data to be captured froma specific direction and/or distance. Any other data input or outputthat can be adjusted based upon a current relative position of a user.

Various other applications, processes and uses are presented below withrespect to the various embodiments.

FIG. 1 illustrates an example situation 100 wherein two persons 102, 104are engaging in a conversation, such as may be part of a businessmeeting or social outing. The first person 102 is viewing the secondperson 104 using an electronic device 106. The device 106 in thisexample has at least one camera on the side facing each person 102, 104,and a display element (not shown) on at least the side facing the firstperson 102. For purposes of simplicity of explanation the first person102 in this example will be referred to as the viewer, and the secondperson 104 will be referred to as the speaker. When viewing the speaker104 using the device, there will be an effective field of view 108through which the viewer can see the speaker 104. This effective fieldof view can correspond to at least a portion of a field of view of atleast one camera on the side of the device 106 facing the speaker 104.

The viewer 102 might want to view the speaker 104 using the device forany of a number of different reasons. For example, the viewer 102 mightbe in a room with several different people, and might want to haveinformation such as the name or title of each person pop up on thescreen when an image of that person is being captured and displayed bythe device. In other cases, such as for gaming or other suchapplications, it might be desirable to provide one or more graphicalimages on top of the image of another person.

For example, in the situation 200 of FIG. 2( a), the speaker 204 isdisplayed using a display screen 208 of an electronic device 202. Aspart of a game, one or more graphical elements 206 are placed on top ofthe image of the speaker 204. This could be, for example, for a personin the game to appear as a particular character in the game. In thisexample, the location of the speaker 204 in the image is determined, andgraphical elements 206 representing a cheap plastic disguise arepositioned in the image to appear as if the speaker is wearing thatdisguise. Approaches for determining the location of a person in animage and for overlaying graphical elements on an image are well knownin the art and will not be discussed in detail herein.

As discussed, it can be desirable in at least certain embodiments toenhance the realism of such a situation as much as possible. Forexample, if the user moves the device and the added graphics are simplycentered relative to the captured image, the changing position of thosegraphics relative to the displayed face will take the viewer out of theexperience, as it will be apparent that the other person is not actuallywearing that disguise. Further, if the image of the speaker changesposition or view as the location or orientation of the phone isadjusted, it will not appear to the user as if there is an actual personacross from that viewer wearing the disguise (i.e., the device will notact as a window through which the viewer is looking).

It would thus be desirable, in at least some embodiments, to adjust thedisplayed image of the speaker 204 such that the view of that speaker,from the perspective of the viewer, does not substantially change as thedevice is rotated or translated. For example, in the situation 210 ofFIG. 2( b) the user has tilted the device along at least one axis. Ascan be seen, however, it still appears as if the view of the speaker 204has not changed. The displayed image thus can appear to the user as ifthe device is a piece of glass, as the displayed image appearsconsistent to the viewer (not including changes due to differences inviewable area of the display, translations of the device, etc.). In atleast some embodiments, a user can rotate, tilt, translate, or otherwisereorient the device in any direction over at least a minimum range ofmovement, and at least the position and/or perspective of the displayedimage will remain substantially consistent from the point of view of theviewer.

As discussed, such ability can provide a type of image stabilization.Using conventional image stabilization approaches, the image as in FIG.2( a) would remain virtually unchanged on the display screen. A purposeof conventional image stabilization approaches is to account for smalltranslations or movements of the device capturing the image, such thatthe image appears on the screen as if the device has not moved and thedisplayed image does not change. Image stabilization algorithms asdiscussed herein, on the other hand, attempt to stabilize the image toaccount (primarily) for movement of the user. If the user does not move,the image seen from the point of view of the user should appear thesame, regardless of movement of the device. Movement of the device wouldin general change the image actually displayed on the device, as theimage would be distorted to compensate for the movement in order toenable the view from the user's perspective to remain substantiallyunchanged. Thus, conventional algorithms attempt to keep the imagedisplayed on the screen consistent, while certain algorithms discussedherein will change the image displayed on the screen in order to keepthe view from the user's perspective consistent.

In order for aspects of the image to appear consistent to the user, theactual image displayed by the device in at least some embodiments ismodified to compensate for any relative changes in position and/ororientation between the viewer and the device. For example, in thesituation 220 of FIG. 2( c) it can be seen that the image of the speaker222 has been stretched along a specific direction. In FIG. 2( b), thedevice has been rotated such that the viewable area (from theperspective of the viewer) of the display element decreased, but thewidth of the speaker did not decrease. In order to provide suchappearance, the device (or a system or service in communicationtherewith) must modify the image based on at least the original imageand the current viewing angle of the viewer. In this example, the imageof the speaker is stretched horizontally to substantially fill thescreen width, such that when viewed at an angle as in FIG. 2( b) theimage appears substantially unchanged from before the rotation.

In another example as illustrated in the situation 300 of FIG. 3( a),the device 302 may not move or change orientation significantly or atall (although such movements are possible), and the primary change inrelative orientation might be based primarily upon movement of theviewer. For example, in FIG. 3( a) the viewer 312 is viewing a hockeygame through an electronic device 302. The viewer and device mightactually be at the hockey rink, or might be viewing a controllable (orother such) feed from the hockey rink. If the viewer is not at the rink,the user can have the ability to obtain different views of the rinkthrough motions of the user. An advantage to viewing the hockey gamethrough the device even if the user is at the rink, however, can be theability of the device to place information over the view of the game.For example, as illustrated in the figure the game displayed 304 canhave several different elements. The viewer 312 might want to easilykeep track of that viewer's favorite player 306 by having the devicedisplay a marker 308 or other such indicator relative to that player.Similarly, in order to more easily be able to determine the currentlocation of the puck, the device might render some type of indicator 310relative to the puck such that the viewer can quickly determine itslocation.

During the game, the viewer 312 might want to view a different portionof the rink. For example, in FIG. 3( a) player 306 has the puck, but theviewer cannot determine from the current view whether anyone is open toreceive a pass. In at least some embodiments, the viewer 312 can shiftposition to the left relative to the device, as illustrated in FIG. 3(b). If the display of the device is still acting like a virtual glasswindow onto the game, the view and/or image displayed can be adjustedbased on the new relative position of the viewer 312.

In the situation 320 of FIG. 3( b), the viewer 312 moving to the lefteffectively causes the image to shift left 328, or the device to “pan”the view 322 to the right, as if the viewer was viewing the game throughan actual window. In this view, an open player 324 can be seen to whomthe puck can be passed. In at least some embodiments, another graphicalindicator 326 can be added to indicate that the player is on the sameteam, open, able to receive the puck, etc.

In some embodiments, the amount of a scene being displayed can varybased at least in part upon how closely the user is positioned relativeto the screen. The user can change the amount of scene displayed by, forexample, pulling the device closer or backing away from the device. Inat least some embodiments, the device can magnify or shrink thedisplayed image to show, for example, more detail or a larger field ofview.

For situations such as that in FIGS. 3( a) and 3(b), there may be littleimage manipulation (e.g., stretching or otherwise deforming) needed dueto the distance to the object being viewed relative to the distance ofthe viewer, although for other situations at least some suchmanipulation may be necessary. There will be a need, however, to atleast change the view displayed by the device. As discussed elsewhereherein, this change in view can be accomplished in any number of ways,such as by using a moving camera, displaying only a portion of the viewof a camera with a wide angle lens, stitching together images frommultiple cameras, etc.

It should be understood that the ability to pan to different portions ofthe game in FIG. 3( b) could also be performed by reorienting the deviceinstead of the user. For example, the viewer could translate andreorient the device such that the device is pointing towards the openplayer, etc. Although this might appear at first glance to utilize adifferent process than was used for the head movement-based adjustment,or the image stabilization of FIG. 2( b), a single algorithm or processcan in fact be used for all these examples.

Consider the example situation 400 of FIGS. 4( a)-4(d). In FIG. 4( a), aviewer 402 is positioned relative to an electronic device 404 such thata primary viewing direction 406 is substantially orthogonal to thedevice. The relative location of the user can be determined using atleast one front-facing camera 410 or other imaging element of thedevice, using any appropriate process as discussed or suggested herein.The primary viewing direction can be determined using any appropriatealgorithm able to determine the direction based at least in part upon adetermined relative position of the user and a position on the device,such as a center point of a display element. As discussed later herein,more precise determinations can be made based upon gaze detection andother such factors. For example, the device can utilize information suchas the distance to the viewer 402, the relative offset of an imagingelement 412 capturing the information, and the field of view 408 of thatimaging element 412 to determine an effective viewing angle (asdiscussed with respect to FIG. 1) of the view through the device.

In FIG. 4( b), the device 406 has been rotated such that the viewingangle 408 of the camera has changed with respect to the viewer 402. Theprimary viewing direction 406 of the viewer, however, has notsubstantially changed. In order to provide consistency of the image fromthe user's point of view, the device has to compensate for theadjustment between the primary viewing direction 406 and the viewingangle 408 of the imaging element. As discussed elsewhere herein, thisadjustment can include, for example, changing the direction of thecamera, selecting a different portion of the field of view, stretchingor compressing the image to compensate for a change in relativeorientation, etc.

In the situation of FIG. 4( c), the viewer 402 has instead shiftedposition relative to the device 404. Although the user would potentiallybe looking at a different view in the display element, it can be seenthat the algorithm essentially handles the movement in the same way, asthe relative orientation of all elements in FIG. 4( c) is the same asthat in FIG. 4( b), and the algorithm would again determine how tocompensate for the change in primary viewing direction 406 with respectto the viewing angle 408 of the imaging element. This can include achange in direction and manipulation of the image, using the sameapproach as in FIG. 4( b). As discussed, in at least some embodimentsthe relative distance of the user with respect to the device can also bedetermined, in order to determine, for example, an amount of detail orzoom level for the displayed image.

A similar approach can be used when a single device is not performingboth the imaging and the relative position determinations. For example,the situation of FIG. 4( d) is similar to that of FIG. 4( c), except inthis situation there is a separate device 414 capturing the imageinformation to be displayed. Here, the primary device 404 can againdetermine the relative position of the viewer 402 to that device, andinformation about the corresponding primary viewing direction 406 can besent to the secondary device 414 in order to correlate that direction tothe viewing angle 408 of the camera of the second device. It should beunderstood that in other embodiments the image information from thesecondary device could instead be sent to the primary device forcomparison with the primary viewing direction, or information from bothdevice can be sent to a remote service, etc. In some embodiments, theactions of the secondary device 414 could instead be performedvirtually, where no actual image is captured but the image informationis rendered from a model, application, etc. As can be seen, thealgorithm would handle the information in essentially the same way, andthe view to the user would be essentially the same, with the onlyprimary difference being the transmission of information to anotherlocation.

FIG. 5 illustrates front and back views of an example computing device500 that can be used in accordance with various embodiments. Although aportable computing device (e.g., a smart phone, an electronic bookreader, or tablet computer) is shown, it should be understood that anydevice capable of receiving and processing input can be used inaccordance with various embodiments discussed herein. The devices caninclude, for example, desktop computers, notebook computers, electronicbook readers, personal data assistants, cellular phones, video gamingconsoles or controllers, television set top boxes, and portable mediaplayers, among others.

In this example, the computing device 500 has a display screen 502,which under normal operation will display information to a user (orviewer) facing the display screen (e.g., on the same side of thecomputing device as the display screen). The computing device in thisexample can include one or more image capture elements, in this exampleincluding two image capture elements 504 on the front of the device andtwo image capture elements 506 on the back of the device, although itshould be understood that additional or fewer image capture elementscould be used, and could also, or alternatively, be placed on the sides,corners, or other locations on the device. The image capture elementsalso can be of similar or different types. Each image capture elementmay be, for example, a camera, a charge-coupled device (CCD), a motiondetection sensor or an infrared sensor, or can utilize other imagecapturing technology. The computing device can also include at least onemicrophone or other audio capture element 508 capable of capturing audiodata, and one or more orientation-determining elements 510, such as anaccelerometer, gyroscope, digital compass, or inertial sensory, that canassist with movement and/or orientation determinations.

FIG. 6 illustrates a set of basic components of a computing device 600such as the device 500 described with respect to FIG. 5. In thisexample, the device includes at least one processor 602 for executinginstructions that can be stored in a memory device or element 604. Aswould be apparent to one of ordinary skill in the art, the device caninclude many types of memory, data storage or computer-readable media,such as a first data storage for program instructions for execution bythe processor 602, the same or separate storage can be used for imagesor data, a removable memory can be available for sharing informationwith other devices, and any number of communication approaches can beavailable for sharing with other devices. The device typically willinclude at least one type of display element 606, such as a touchscreen, electronic ink (e-ink), organic light emitting diode (OLED) orliquid crystal display (LCD), although devices such as portable mediaplayers might convey information via other means, such as through audiospeakers. As discussed, the device in many embodiments will include atleast two image capture elements 608, such as at least one image captureelement positioned to determine a relative position of a viewer and atleast one image capture element operable to image a user, people, orother viewable objects in the vicinity of the device. An image captureelement can include any appropriate technology, such as a CCD imagecapture element having a sufficient resolution, focal range and viewablearea, to capture an image of the user when the user is operating thedevice. Methods for capturing images or video using an image captureelement with a computing device are well known in the art and will notbe discussed herein in detail. It should be understood that imagecapture can be performed using a single image, multiple images, periodicimaging, continuous image capturing, image streaming, etc.

The device can include at least one orientation determining element 610,such as an accelerometer, digital compass, electronic gyroscope, orinertial sensor, which can assist in determining movement or otherchanges in orientation of the device. The device can include at leastone additional input device 612 able to receive conventional input froma user. This conventional input can include, for example, a push button,touch pad, touch screen, wheel, joystick, keyboard, mouse, trackball,keypad or any other such device or element whereby a user can input acommand to the device. These I/O devices could even be connected by awireless infrared or Bluetooth or other link as well in someembodiments. In some embodiments, however, such a device might notinclude any buttons at all and might be controlled only through acombination of visual and audio commands such that a user can controlthe device without having to be in contact with the device.

FIG. 7 illustrates an example of a first portion 700 of a process forproviding a relative orientation-based image display in accordance withvarious embodiments. It should be understood that, for any processdiscussed herein, there can be additional, fewer, or alternative stepsperformed in similar or alternative orders, or in parallel, within thescope of the various embodiments unless otherwise stated. In thisexample, position tracking of a viewer is activated on the device 702.In some embodiments a user must activate this mode manually, while inother modes the device can activate the mode automatically when a personis detected nearby. Other modes of activation are possible as well, suchas upon a user opening a specific application on the device. When theposition tracking is active, the device can begin imaging around thedevice 704, whether in all directions, some directions, a specific rangeof directions, or a direction substantially toward a determined viewer.As discussed elsewhere herein, in some embodiments the imaging willinvolve ambient light image or video capture, while other embodimentscan utilize infrared imaging, heat signature detection, or any othersuch approach. The device can analyze the captured image information toattempt to locate facial features of a user 706, or at least a personnearby, where those features in some embodiments include at least theeyes of a user. In some embodiments, the device will attempt to locatean object that is shaped like a human head and that contains twoeye-like features. In other embodiments, facial recognition or any othersuch algorithm can be used to attempt to determine the presence of ahuman head, or other portion or feature of a user, in the field of viewof at least one of the imaging elements.

Once the facial features are located, the device can attempt todetermine aspects or information relating to those features 708. In thisexample, the determined aspects can be used to attempt to determine arelative orientation between the device and the user, as well as theorientation of those features relative to the device 710, which can beuseful in determining information such as the present viewing locationof a user. The determined aspects then can be monitored over time 712,such as by continuing to capture and analyze image information todetermine the relative position of the user and/or orientation of thedevice. A change in the aspect, such as a change in position ororientation, can be determined 714, and the device can determine whetherthat change requires an adjustment to the image to be displayed 716. Forexample, a rotating the device or shifting position might require thedisplayed image to be shifted or manipulated to maintain consistencyfrom the point of view of the viewer. If the orientation change issufficient to warrant an adjustment, the device can determine andperform the appropriate adjustment to the image information 718, such asto stretch, compress, pan, or rotate the image.

As an example of such adjustment, FIG. 8 illustrates a second portion800 of a process for modifying the image in response to a determinedchange in orientation that can be used in accordance with variousembodiments. In this example, aspects of the image and any otherinformation displayed can be based at least in part upon the relativeposition of a viewer, although various other approaches could be used aswell as discussed elsewhere herein. During operation, an electronicdevice can determine (and monitor over time) the relative position aviewer 802. Based at least in part upon the relative position, a primaryviewing direction or viewing angle of the viewer can be determined 804,the device can determine the appropriate portion of the camera view thatcorresponds to that angle 806. As discussed, the camera view mightrequire translating the camera, capturing image information from adifferent camera, or selecting a different portion of a larger imagecaptured by the camera.

The primary viewing direction and relevant portion of the camera viewthen can be compared to the currently and/or previously displayed imageto determine whether any movement or change in relative orientation hasoccurred that would result in a detectable change in the image from theperspective of the viewer 808. If so, the image information can bemanipulated (e.g., stretched, compressed, translated, etc.) to maintainconsistency of view as discussed elsewhere herein 810. After anynecessary manipulation has been made, the relevant portion of the imageinformation from the appropriate viewing direction can be displayed onthe device 812.

Various approaches can be utilized for locating one or more desiredfeatures of a user's face to determine various aspects useful fordetermining relative orientation. For example, an image can be analyzedto determine the approximate location and size of a user's head or face.FIG. 9( a) illustrates an example wherein the approximate position andarea of a user's head or face 900 is determined and a virtual “box” 902is placed around the face as an indication of position using one of aplurality of image analysis algorithms for making such a determination.Using one algorithm, a virtual “box” is placed around a user's face andthe position and/or size of this box is continually updated andmonitored in order to monitor relative user position. Similar algorithmscan also be used to determine an approximate location and area 904 ofeach of the user's eyes (or in some cases the eyes in tandem). Bydetermining the location of the user's eyes as well, advantages can beobtained as it can be more likely that the image determined to be theuser's head actually includes the user's head, and it can be determinedthat the user is facing the device. Further, the relative movement ofthe user's eyes can be easier to detect than the overall movement of theuser's head when performing motions such as nodding or shaking the headback and forth. Monitoring box size also helps to provide distanceinformation as well as directional information, which can be helpfulwhen generating a three-dimensional model for modifying imageinformation based on relative user position.

Various other algorithms can be used to determine the location offeatures on a user's face. For example, FIG. 9( b) illustrates anexample wherein various features on a user's face are identified andassigned a point location 906 in the image. The system thus can detectvarious aspects of user features and can determine more subtle changesin orientation. Such an approach provides advantages over the generalapproach of FIG. 9( a) in certain situations, as various other featurescan be determined, in case the user's eyes cannot be seen due toglasses, hair, etc.

Once the positions of facial features of a user are identified, relativemotion between the user and the device can be detected and utilized asinput. For example, FIG. 9( c) illustrates an example where the user'shead 900 is moving up and down with respect to the viewable area of theimaging element. As discussed, this could be the result of the usermoving his or her head, or the user moving the device up and down, etc.FIG. 9( d) illustrates a similar example wherein the user is movingright to left relative to the device, through movement of the user, thedevice, or both. As can be seen, each movement can be tracked as avertical or horizontal movement, respectively, and each can be treateddifferently as an input to modify a displayed image. As should beunderstood, such a process also can detect diagonal or other suchmovements. FIG. 9( e) further illustrates an example wherein the usertilts the device and/or the user's head, and the relative change in eyeposition is detected as a rotation. In some systems, a “line” thatcorresponds to the relative position of the eyes can be monitored, and ashift in angle of this line can be compared to an angle threshold todetermine when the rotation should be interpreted as input. FIG. 9( f)illustrates another advantage of using an approach such as thatdescribed with respect to FIG. 9( b) to determine the position ofvarious features on a user's face. In this exaggerated example, it canbe seen that the features of a second user's head 1008 have a differentrelative position and separation. Thus, the device also can not onlydetermine positions of features for a user, but can distinguish betweendifferent users.

FIGS. 10( a) and 10(b) illustrate an example approach that can be usedto determine variations in relative distance between a user and a devicethat can be used in accordance with various embodiments. As in FIG. 9(a), the approximate position and area of a user's head or face 1000 isdetermined and a virtual “box” 1002 is placed around the face at aninitial distance as an indication of distance using one of a pluralityof image analysis algorithms for making such a determination. If theuser is known, the size of the user's head might be stored such that anactual distance to the user can be calculated based at least in partupon the size of the box 1002. If the user is not known, the distancecan be estimated or determined using other factors, such as stereoscopicimaging. In some embodiments, determinations will be relative withrespect to an initial box size when the actual distance cannot bedetermined.

As the distance between the user and the device changes, the size of thevirtual box will change as well. For example, in FIG. 10( b) thedistance between the user and the device has increased, such that theuser's head 1020 appears smaller in the captured image information.Accordingly, the size of the virtual box 1022 for the adjusted size ofthe user's head is smaller than the original box 1002 for the initialdistance. By monitoring adjustments in the size of the box or anothermeasure of the user's head and/or other such features (e.g., boxes1024), the device can determine an approximate distance and/or change indistance to the user. As discussed, this information can be used toadjust aspects of the displayed image information such as a level ofzoom or amount of detail.

In some embodiments, a computing device can determine and track anapproximate area or region of interest corresponding to the user's eyes,or another such feature, in the captured images such that an algorithmof the computing device only has to analyze image data corresponding tothat region, which can significantly reduce the amount of processingneeded for images, particularly for high resolution, full color images.

A number of other approaches can be used as well within the scope of thevarious embodiments. For example, thermal imaging or another suchapproach could be used to attempt to determine and track the position ofat least some aspect of a human user. In many instances the imagingsystem is desired to be small and inexpensive enough for mass marketing,such that simple or conventional imaging approaches and components canbe preferred.

As mentioned, it can be desirable in at least some embodiments toutilize at least two imaging elements (i.e., stereoscopic imaging) todetermine the location of the user, as well as to capture imageinformation to be displayed. In almost all situations the position of animaging element will be offset from the eye of a user, such that someimage translation and viewing angle adjustments may need to be made toensure the consistency of the displayed image. Particularly forapplications such as image stabilization from the point of view of theviewer, it can be important to compensate for differences in viewingangle resulting from the camera being offset from the user's eye.

An imaging element capturing image information in at least oneembodiment is a relatively high resolution camera that is able tocapture image information over a large angular range. The offset of theviewer's eyes to the imaging element can create a parallax effect thatwill cause the window effect to be less than ideal in many situations.For example, referring back to FIG. 4( a) the viewing angle from theviewer 402 to the end of arrow 406 will be slightly different than theviewing angle from imaging element 412 because there is an offset therebetween. In order to compensate for the offset such that the view of theworld from the camera matches the view of the world by the viewer, athree-dimensional model of the surrounding area can be generated thatenables the device to rotate the view as necessary to match the view ofthe user. Without stereo cameras the device can attempt to minimize theeffects of parallax by translating the image or making othermanipulations, but there will be some slight differences due to theoffset.

A number of different applications can take advantage of variousapproaches discussed herein in accordance with the various embodiments.For example, an electronic device might be placed on a tripod to recordan activity. If a viewer is moving his or her head to see the portion ofthe activity that is of interest, the device can record that portion ofthe image that is displayed as a result of orientation changes. Such anapproach enables a viewer to record certain portions of the activity,such as that viewer's child in a game. Such an approach also enablesinformation to be added to the image as discussed above.

For example, a viewer in a store might be able to view one or moreproducts through the device and see information (e.g., reviews,description, pricing, etc.) about that product displayed relative to theimage of the device. As the viewer changes orientation and differentproducts come into view, the added information can update accordingly.In some embodiments, a viewer can also obtain information aboutdifferent versions of the product, compatible accessories, or other suchinformation.

In one embodiment, image recognition can be used with the imagingprocess to identify specific people, objects, or other such things. Forexample, a viewer might be looking for a person in a crowd. The viewercould look “through” the device while scanning the device across thecrowd, and when the desired person is recognized by the device as cominginto view the device can generate a notification for the viewer, such asadding graphics or text to the image, making an audible noise, causingthe device to vibrate, or performing another such notification action.

As discussed, in various embodiments a device can utilize the viewer'sgaze direction as an input as well as the relative position. Theusefulness of such information can increase with an increase in the sizeof the display for at least some applications. For example, if thedisplay is a large screen television display then the user might only beable to focus on a portion of the screen at any time. Thus, the portionat which the viewer is looking can provide certain functionality. Forexample, the device can record portions of a video that are most ofinterest to the viewer. For example, if the viewer is watching a childplay soccer then the device can record a portion of the overall imagethat is displayed but that corresponds to the viewer's interest, whichwill generally be primarily directed towards the child. If theresolution is sufficient, the viewer then can obtain a video file thatis “zoomed in” on the portion of interest to that viewer during the timeof the recording. If there are instead multiple viewers around thetelevision and only one of the viewers wants additional information suchas statistics or other information overlaid, the device can cause theadditional information to only be displayed in the portion where theviewer is looking, leaving the rest of the image free of suchinformation.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 11 illustrates an example of an environment 1100 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes an electronic client device 1102, whichcan include any appropriate device operable to send and receiverequests, messages or information over an appropriate network 1104 andconvey information back to a user of the device. Examples of such clientdevices include personal computers, cell phones, handheld messagingdevices, laptop computers, set-top boxes, personal data assistants,electronic book readers and the like. The network can include anyappropriate network, including an intranet, the Internet, a cellularnetwork, a local area network or any other such network or combinationthereof. Components used for such a system can depend at least in partupon the type of network and/or environment selected. Protocols andcomponents for communicating via such a network are well known and willnot be discussed herein in detail. Communication over the network can beenabled via wired or wireless connections and combinations thereof. Inthis example, the network includes the Internet, as the environmentincludes a Web server 1106 for receiving requests and serving content inresponse thereto, although for other networks, an alternative deviceserving a similar purpose could be used, as would be apparent to one ofordinary skill in the art.

The illustrative environment includes at least one application server1108 and a data store 1110. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to any deviceor combination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 1108 caninclude any appropriate hardware and software for integrating with thedata store 1110 as needed to execute aspects of one or more applicationsfor the client device and handling a majority of the data access andbusiness logic for an application. The application server providesaccess control services in cooperation with the data store and is ableto generate content such as text, graphics, audio and/or video to betransferred to the user, which may be served to the user by the Webserver 1106 in the form of HTML, XML or another appropriate structuredlanguage in this example. The handling of all requests and responses, aswell as the delivery of content between the client device 1102 and theapplication server 1108, can be handled by the Web server 1106. Itshould be understood that the Web and application servers are notrequired and are merely example components, as structured code discussedherein can be executed on any appropriate device or host machine asdiscussed elsewhere herein.

The data store 1110 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 1112 and userinformation 1116, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 1114. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 1110. The data store 1110 is operable,through logic associated therewith, to receive instructions from theapplication server 1108 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on the user device 1102. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 11. Thus, the depiction of the system 1100 in FIG.11 should be taken as being illustrative in nature and not limiting tothe scope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A method of providing position-dependent image display, comprising: under control of one or more computing systems configured with executable instructions, capturing at least one first image of a viewer of a display element on an electronic device using at least one first imaging element of the electronic device; based at least in part upon the at least one first image, determining a relative orientation of the viewer with respect to the electronic device; capturing at least one second image using at least one second imaging element of the electronic device, the at least one second image including at least image information from a viewable area of the viewer as obstructed by at least a portion of the electronic device; determining a final portion of the at least one second image corresponding to the obstructed region of the viewable area; based at least in part upon the relative orientation of the viewer with respect to the electronic device, performing any manipulation of the final portion of the at least one second image such that the final portion is displayed to the viewer with a perspective corresponding substantially to the perspective of the final portion if the electronic device was not obstructing the obstructed region; and displaying the final portion after any manipulation to the viewer using a display element of the electronic device, wherein the perspective of an image being captured and displayed by the electronic device is substantially maintained from the point of view of the viewer regardless of changes in orientation of the viewer with respect to the electronic device over at least a viewable range of motion.
 2. The method of claim 1, wherein the at least one first image of the viewer is captured using at least two imaging elements to determine a three-dimensional relative position of the viewer with respect to the electronic device.
 3. The method of claim 1, wherein the at least one second image is captured using at least one two imaging elements to generate a three-dimensional model of at least the viewable area obstructed by at least a portion of the electronic device.
 4. The method of claim 1, wherein performing any manipulation of the final portion of the at least one second image includes utilizing the three-dimensional model to offset parallax due to a relative position of the at least one second imaging element with respect to the viewer.
 5. The method of claim 1, wherein any adjustment to account for parallax is approximated for two-dimensional data.
 6. The method of claim 1, wherein the at least one second imaging element capturing at least one second image is associated with a remote electronic device, the manipulation of the final portion of the at least one second image being such that the final portion is displayed to the viewer with a perspective corresponding substantially to the perspective of the final portion if the remote electronic device was substantially at the position of the electronic device and the remove electronic device was not obstructing the obstructed region.
 7. A method of displaying an image on an electronic device, comprising: under control of one or more computing systems configured with executable instructions, capturing image information of a user of the electronic device; determining a relative orientation between at least one feature of the viewer of the electronic device and the electronic device based at least in part upon the captured image information; determining a viewable area of an image to be displayed on the electronic device corresponding to the relative orientation of the viewer; and performing any manipulation of the viewable area of the image, based at least in part upon the determined relative orientation, to enable at least a portion of the image to maintain substantially the same perspective from the point of view of the viewer, when displayed by the electronic device, regardless of changes in the relative orientation between the viewer of the electronic device over at least a range of changes in relative orientation.
 8. The method of claim 7, wherein the changes in relative orientation are capable of being performed through at least one of movement of the viewer or movement of the electronic device.
 9. The method of claim 7, wherein the image to be displayed is captured using at least one image capture element of the electronic device or rendered by the electronic device.
 10. The method of claim 7, wherein at least a portion of the image further appears in substantially the same position from the point of view of the viewer, when displayed by the electronic device, regardless of changes in a relative position between the viewer of the electronic device over at least a range of changes in relative position.
 11. The method of claim 7, further comprising: analyzing at least one object in the image to be displayed to determine identifying information about the at least one object.
 12. The method of claim 11, further comprising: overlaying content on the at least one object when displayed on the electronic device, the content relating to the identifying information.
 13. The method of claim 12, wherein the content remains positioned over the at least one object as the position of the object moves in the image displayed on the electronic device.
 14. The method of claim 11, wherein an apparent position of the overlaid content is maintained in three-dimensional space from the point of view of the viewer.
 15. The method of claim 7, wherein the image displayed on the electronic device is displayed using a three-dimensional display to minimize focusing issues of the human eye.
 16. The method of claim 7, further comprising: determining a gaze direction of the viewer; and modifying at least a portion of the displayed image based at least in part upon the determined gaze direction.
 17. The method of claim 7, wherein determining the relative orientation between the viewer and the electronic device includes using at least two imaging elements to determine a three-dimensional relative position of the viewer with respect to the electronic device.
 18. The method of claim 7, wherein determining a relative orientation between a viewer of the electronic device and the electronic device includes capturing image information for the viewer using at least one of thermal imaging, infrared radiation detection, and motion detection.
 19. A computing device, comprising: a processor; a display element; at least two image capture elements; and a memory device including instructions operable to be executed by the processor to perform a set of actions, enabling the computing device to: determine a relative orientation between a viewer of the computing device and the computing device using at least one of the image capture elements; determine a viewable area of an image to be displayed on the display element corresponding to the relative orientation of the viewer; perform any manipulation of the viewable area of the image, based at least in part upon the determined relative orientation, to enable at least a portion of the image to maintain substantially the same perspective from the point of view of the viewer, when displayed by the electronic device, regardless of changes in the relative orientation between the viewer of the electronic device over at least a range of changes in relative orientation.
 20. The computing device of claim 19, wherein the changes in relative orientation are capable of being performed through at least one of movement of the viewer or movement of the computing device.
 21. The computing device of claim 19, wherein the relative orientation between the viewer and the electronic device is determined at least in part by capturing at least one image of the viewer using at least one of the image capture elements of the electronic device and analyzing the at least one image to determine the relative orientation.
 22. The computing device of claim 19, wherein at least a portion of the image further appears in substantially the same position from the point of view of the viewer, when displayed by the computing device, regardless of changes in a relative position between the viewer of the computing device over at least a range of changes in relative position.
 23. The computing device of claim 19, wherein the image to be displayed is captured using at least two of the image capture elements, and wherein the instructions further enable the computing device to generate a three-dimensional model of the image to be used in performing any manipulation of the image based on the relative position of the viewer.
 24. A non-transitory computer-readable storage medium storing processor-executable instructions for controlling a computing device, comprising: program code for determining a relative orientation between a viewer of the computing device and the computing device using at least one of the image capture elements; program code for determining a viewable area of an image to be displayed on the display element corresponding to the relative orientation of the viewer; program code for performing any manipulation of the viewable area of the image, based at least in part upon the determined relative orientation, to enable at least a portion of the image to maintain substantially the same perspective from the point of view of the viewer, when displayed by the electronic device, regardless of changes in the relative orientation between the viewer of the electronic device over at least a range of changes in relative orientation.
 25. The non-transitory computer-readable storage medium of claim 24, wherein the changes in relative orientation are capable of being performed through at least one of movement of the viewer or movement of the computing device.
 26. The non-transitory computer-readable storage medium of claim 24, wherein the relative orientation between the viewer and the electronic device is determined at least in part by capturing at least one image of the viewer using at least one of the image capture elements of the electronic device and analyzing the at least one image to determine the relative orientation.
 27. The non-transitory computer-readable storage medium of claim 24, wherein the image to be displayed is captured using at least two of the image capture elements, and wherein the instructions further enable the computing device to generate a three-dimensional model of the image to be used in performing any manipulation of the image based on the relative position of the viewer. 